What Should We Do in the Absence of Evidence?

Pictured: Smarter than you.

For every complex problem, there is an answer that is clear, simple—and wrong.

– H.L. Mencken

Despite my multiple personalities, it seems that only the OCD doctor gets anything done. The Goth cowgirl persona? Lazy. And the NBA playoffs are sucking up an inordinate amount of time. Go Blazers. Just not very far. Sigh. But what are you going to do. Work needs doing and someone has to do it.

This week was one of deadlines. In June I am giving a series of talks at the SMACC conference in Chicago and I have to have all my talks ready to go today. So sometimes to meet all my deadlines I need to re-purpose other material.

Spoiler alert: if you are going to be at SMACC and hear my lectures, stop reading here. Everything I am going say in 6 weeks will follow. And really even if you are going to SMACC, it is a content-free post. You might be better off spending your time elsewhere.

My topic is “What Should We Do in the Absence of Evidence?” It is often said that only 15% of medical practice is evidenced based. It is a myth, and an outdated myth at that, based on a survey from 1961. Like all myths it has staying power and people like to repeat the factoid without bothering to see if it is actually true. It isn’t.

The real data suggests we do a pretty good job at following evidence as long as we are not TV doctors:

Thus, published results show an average of 37.02% of interventions are supported by RCT (median = 38%). They show an average of 76% of interventions are supported by some form of compelling evidence (median = 78%).

There is variability depending on the specialty and how you judge the evidence. Anesthesiologists do the best, but then they have the least to do.

There is always evidence. There are more than 23,000,000 citations on Pubmed and 799,000,000 hits searching ‘medicine’ on Google. It is not a lack of evidence that is the problem. It is more likely that the practitioner doesn’t ww the evidence or the evidence may suck.

I tend to get consults for odd or unusual infections for which there is little data beyond case reports and the antibiotic susceptibility. So I am used to dealing with supporting data that is sparse at best. I often say that part of being a specialist is being ignorant with style.

So what to do.

First. Don’t panic. And carry a towel. Most of the time there is no hurry to intervene in whatever process is making the patient ill. You usually have time to sort things out. As the saying goes,

AT A CARDIAC ARREST, THE FIRST PROCEDURE IS TO TAKE YOUR OWN PULSE.

Most of the pressure comes not from the patient and her disease but the systems that do not allow the slow and methodical evaluation of disease. The old definition of a Fever of Unknown Origin, two weeks in the hospital with a fever, is a fantasy in modern medicine. Madeline would have her admission denied.

So take your time. Watch and wait. Much of the time when you do nothing, eventually the answer will become obvious.

THE DELIVERY OF GOOD MEDICAL CARE IS TO DO AS MUCH NOTHING AS POSSIBLE.

Yet another proverb:

Muddy water. Let stand. Will clear.

The “don’t just stand there, do something” that drives an intern becomes “don’t just do something, stand there” that I specialize in. And my wife can vouch for me: I am most excellent at doing nothing.

Besides, I learned as a resident that if you have no data to support an intervention, you are more likely to cause harm than to do good. I have no data to support this assertion and have never found a reference, so it is one of innumerable bits of unvalidated ‘wisdom’ I have picked up over the years. Medicine should be neither Nike (just do it) nor a Chimp (throwing crap at the wall and seeing what sticks).

Often the intervention is uncertain because you do not have a diagnosis. The best thing to be done is to retake the history, looking for hints in the pattern of disease or other risks for odd diseases. Then Google and Pubmed everything, especially the odder aspects of the case. It is amazing how often Google will come up with the right diagnosis, or at least point you in the right direction. And once you have the diagnosis, then you can proceed with the right therapy.

But be skeptical. It is easy to become enamored of a diagnosis and treatment. And do not let advertising terms guide therapy. There are only four absolutes in medicine with no exceptions. Ever. One absolute is anyone who routinely wears their scrubs outside the hospital is a dick. Another is anyone who uses “strong”, “big gun”, or “powerful” as an adjective about antibiotics is an idiot who knows nothing about the treatment of infections. They are advertising adjectives, providing false comfort but no benefit. Don’t do an intervention to make yourself feel good. It really is OK to say “I don’t know what to do” as long as you have a plan with how to proceed.

But let’s say you have a problem for which there is uncertainty as to the proper intervention. There are no guidelines or your particular patient doesn’t match the patient population in the studies. When I was a resident, all the data for cardiovascular treatments appeared to come from VA studies of old, white smoking males. Do the results apply to other populations? Maybe. Maybe not.

Now what?

You can reason from basic principles of anatomy, physiology etc. In ID it is kinda sorta easy: if the drug kills the organism in the test tube and then gets into the infected space, it should work. Other diseases are not so easy. When in doubt, should the disease get steroids? There is probably no disease, including Cushing’s Syndrome, for which steroids are considered by someone despite a lack of supporting data.

But at the end of the day you need to fire up the Chrome and go looking for the evidence, such as it may be.

So then you have to consider the published literature. What are the levels of evidence?

Eminence based medicine—The more senior the colleague, the less importance he or she placed on the need for anything as mundane as evidence. Experience, it seems, is worth any amount of evidence. These colleagues have a touching faith in clinical experience, which has been defined as “making the same mistakes with increasing confidence over an impressive number of years.” The eminent physician’s white hair and balding pate are called the “halo” effect.

Vehemence based medicine—The substitution of volume for evidence is an effective technique for brow beating your more timorous colleagues and for convincing relatives of your ability.

Eloquence based medicine—The year round suntan, carnation in the button hole, silk tie, Armani suit, and tongue should all be equally smooth Sartorial elegance and verbal eloquence are powerful substitutes for evidence.

Providence based medicine—If the caring practitioner has no idea of what to do next, the decision may be best left in the hands of the Almighty. Too many clinicians, unfortunately, are unable to resist giving God a hand with the decision making.

Diffidence based medicine—Some doctors see a problem and look for an answer. Others merely see a problem. The diffident doctor may do nothing from a sense of despair. This, of course, may be better than doing something merely because it hurts the doctor’s pride to do nothing

Nervousness based medicine—Fear of litigation is a powerful stimulus to over investigation and over treatment. In an atmosphere of litigation phobia, the only bad test is the test you didn’t think of ordering.

Confidence based medicine—This is restricted to surgeons.

There many conceptual frameworks that ranks levels of evidence in medicine:

Level Type of evidence

1A Systematic review (with homogeneity) of RCTs

1B Individual RCT (with narrow confidence intervals)

1C All or none study

2A Systematic review (with homogeneity) of cohort studies

2B Individual Cohort study (including low quality RCT, e.g. <80% follow-up)

2C “Outcomes” research; Ecological studies

3A Systematic review (with homogeneity) of case-control studies

3B Individual Case-control study

4 Case series (and poor quality cohort and case-control study

5 Expert opinion without explicit critical appraisal or based on physiology bench research or “first principles”

Which works well when dealing with reality-based processes, but as this blog has noted from the beginning, it does not work for SCAMs when there is no prior plausibility for the intervention.

Of course the volume of information makes it virtually impossible for an individual clinician to master the literature on the odd disease that may appear in the ER or on the ward. It is difficult to read and appreciate a paper above your pay grade. I can read an ID paper with reasonable comfort, but although I am board-certified in Internal Medicine, a paper on cardiology or nephrology can be difficult to place in context. I have to use rules to help me understand it:

• Research Phrase: It has long been known
  Translation/Meaning: I didn’t look up the original reference.

• Research Phrase: A definite trend is evident
  Translation/Meaning: These data are practically meaningless.

• Research Phrase: Of great theoretical and practical importance
  Translation/Meaning: Interesting to me.

• Research Phrase: While it has not been possible to provide definite answers to these questions
  Translation/Meaning: An unsuccessful experiment, but I still hope to get it published.

• Research Phrase: Three of the samples were chosen for detailed study
  Translation/Meaning: The results of the others didn’t make any sense.

• Research Phrase: Typical results are shown
  Translation/Meaning: This is the prettiest graph.

• Research Phrase: These results will be shown in a subsequent report
  Translation/Meaning: I might get around to this sometime, if I’m pushed / funded.

• Research Phrase: The most reliable results are those obtained by Jones
  Translation/Meaning: He was my graduate assistant.

• Research Phrase: It is believed that
  Translation/Meaning: I think.

• Research Phrase: It is generally believed that
  Translation/Meaning: A couple of other people think so, too.

• Research Phrase: It is clear that much additional work will be required before a complete understanding of the phenomenon occurs
  Translation/Meaning: I don’t understand it.

• Research Phrase: Correct within an order of magnitude
  Translation/Meaning: Wrong.

• Research Phrase: In my experience
  Translation/Meaning: Once.

• Research Phrase: In case after case
  Translation/Meaning: Twice.

• Research Phrase: In a series of cases
  Translation/Meaning: Thrice.

• Research Phrase: According to statistical analysis
  Translation/Meaning: Rumor has it.

• Research Phrase: A statistically oriented projection of the significance of these findings
  Translation/Meaning: A wild guess.

• Research Phrase: Thanks are due to Joe Blotz for assistance with the experiment and to George Frink for valuable discussions
  Translation/Meaning: Blotz did the work and Frink explained to me what it meant.

• Research Phrase: A careful analysis of obtainable data
  Translation/Meaning: Three pages of notes were obliterated when I knocked over a glass of wine.

• Research Phrase: It is hoped that this study will stimulate further investigation in this field
  Translation/Meaning: I quit.

In the end your best bet is to call someone like me. Someone who, within the limitations of life, has a (relative) mastery of the topic. But more importantly, it then becomes someone else’s problem to figure out what to do. Good luck. You will need it.